Skip to content

CI: harden build-wheel.yml for the eMudhra/emSign CA incident (13.4.x backport) - #3305

Merged
juenglin merged 4 commits into
NVIDIA:13.4.xfrom
juenglin:emudhra-build-wheel-13.4.x
Nov 5, 2026
Merged

juenglin merged 4 commits into
NVIDIA:13.4.xfrom
juenglin:emudhra-build-wheel-13.4.x

Conversation

@juenglin

@juenglin juenglin commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Description

Refs #3195

Backports the CI hardening that main received for the eMudhra/emSign CA key compromise (CVE-2026-111943, CVE-2026-115557) to 13.4.x, so that wheels released from this branch are built with patched tooling and verified downloads. Before this change the branch's cibuildwheel builds used the pip frontend with an unpatched pip, installed an unverified releases/latest sccache, and downloaded CTK archives and GPU driver installers without checksums.

  • install-sccache action: pin rapidsai/sccache v0.18.0-rapids.7 and verify its sha256 (CI: pin and verify sccache via a shared composite action #3248), and use it in test-sdist-linux.yml instead of mozilla-actions/sccache-action.
  • GPU driver installers: require and verify DRIVER_SHA256 before running the installer, with a test-matrix check (CI: verify GPU driver installer SHA-256 before execution #3249).
  • upgrade-ca-certificates action: run in build-wheel.yml, coverage.yml, test-sdist-windows.yml and the Windows pre-commit job in ci.yml (CI: add eMudhra/emSign CA stopgap for GitHub-hosted Windows runners #3270).
  • setup-hardened-python action: replaces actions/setup-python plus harden-pip everywhere, including the release verification jobs (CI: install eMudhra-fixed Pythons with a pinned uv #3275). This branch still builds and tests Python 3.10, which has no fixed release, so for series older than 3.11 the action warns and installs the sha256-pinned pip 26.2.2 wheel instead of failing.
  • cibuildwheel uses build[uv] with UV_* constraints and the hardened host Python, as main does (build: switch to uv as build frontend #2964). This drops the dummy wheel test stage (sccache stats are dumped during repair) and the before-build = "pip install delvewheel" overrides in the pyproject.toml files.
  • fetch_ctk: verify the CTK redistrib JSON against a pinned sha256 and each component archive against the sha256 in that JSON before extracting it, failing closed on a mismatch. This adds component-sha256 and verify-metadata to ci/tools/fetch_ctk_redistrib.py, with unit tests. All CUDA versions used by this branch's CI (12.6.3, 12.9.1, 13.0.2, 13.4.2) are pinned; unpinned versions only warn. This part is not on main yet.

Not backported: the runner trust probe, the "Enable Git symlinks" steps, and the lychee and pre-commit.yml workflows, which do not exist on this branch.

Notes for reviewers

The commit is a squash of cherry-picks with adaptations for this branch; the conflicts were in the workflow files, which differ from main (cuda_bindings/ instead of cuda_bindings_13/, no lychee workflows, Python 3.10 still supported).

Validation: ci/tools/tests pass locally (79 tests), pre-commit including actionlint passes, and the pinned redistrib hashes were re-checked against the live JSONs. The workflow changes have not run on GitHub Actions yet, so CI on this PR is the real test, in particular the Windows cibuildwheel jobs with build[uv] and the Python 3.10 fallback.

Checklist

  • I have read the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

Summary by CodeRabbit

  • Reliability
    • Build and test workflows now verify downloaded CUDA components and GPU driver installers against expected SHA-256 checksums before use.
    • Linux and Windows workflows now check and update certificate trust where applicable.
  • Builds
    • Build workflows now use a consistent managed Python setup and updated caching for selected Linux builds.
  • Maintenance
    • Added checks to validate driver version hashes used in test configurations.

… backport)

Backport the CI hardening that main received for the eMudhra/emSign CA key
compromise (CVE-2026-111943, CVE-2026-115557), so that wheels released from
this branch are built with patched tooling and verified downloads:

- install-sccache: pin rapidsai/sccache v0.18.0-rapids.7 and verify its
  sha256 instead of piping releases/latest into sudo tar (NVIDIA#3248). Also use
  it in test-sdist-linux.yml instead of mozilla-actions/sccache-action.
- GPU driver installers: verify DRIVER_SHA256 before running them as
  root/admin, with a test-matrix check (NVIDIA#3249).
- upgrade-ca-certificates: upgrade ca-certificates on GitHub-hosted Linux
  and distrust the compromised intermediates on GitHub-hosted Windows, in
  build-wheel.yml, coverage.yml, test-sdist-windows.yml and the Windows
  pre-commit job in ci.yml (NVIDIA#3270).
- setup-hardened-python: install interpreters with a pinned uv and fail on
  unpatched ones, replacing actions/setup-python + harden-pip (NVIDIA#3275). This
  branch still builds and tests Python 3.10, which has no fixed release; for
  series older than 3.11 the action warns and installs pip from the
  sha256-pinned 26.2.2 wheel instead of failing. cibuildwheel now runs with
  the hardened host Python (`python-path`).
- cibuildwheel: use build[uv] with UV_* constraints, as main does (NVIDIA#2964),
  so build dependencies are not fetched by the pip that cibuildwheel
  provides. Drop the dummy wheel test stage (sccache stats are dumped during
  repair) and the `before-build = "pip install delvewheel"` overrides.
- fetch_ctk: verify the CTK redistrib JSON against a pinned sha256 and every
  component archive against the sha256 in that JSON before extracting it,
  failing closed on a mismatch (new `component-sha256` and `verify-metadata`
  subcommands in ci/tools/fetch_ctk_redistrib.py, with tests).

Not backported: the runner trust probe, the "Enable Git symlinks" steps, and
the lychee/pre-commit workflows, which do not exist on this branch.

Refs NVIDIA#3195
@juenglin juenglin added the CI/CD CI/CD infrastructure label Oct 30, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@juenglin juenglin self-assigned this Oct 30, 2026
@github-actions github-actions Bot added the cuda.core Everything related to the cuda.core module label Oct 30, 2026
@juenglin

Copy link
Copy Markdown
Contributor Author

/ok to test 3541ec5

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor
Doc Preview CI
Preview removed because the pull request was closed or merged.

@juenglin

juenglin commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Action failed

Review failed.

@NVIDIA NVIDIA deleted a comment from coderabbitai Bot Nov 5, 2026
@juenglin

juenglin commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 38fc040

@juenglin
juenglin marked this pull request as ready for review November 5, 2026 16:16
…ntainer

Two follow-ups from CodeRabbit review of the eMudhra hardening backport:

- fetch_ctk: add 'verified' prefix to CTK_CACHE_KEY / CTK_CACHE_FILENAME so
  that cache entries written before the hash-verification logic was introduced
  cannot hit and bypass the new verify-metadata / archive SHA-256 checks.

- test-wheel-linux: upgrade ca-certificates inside the ubuntu:22.04 test
  container before its first HTTPS operation. The host-level
  upgrade-ca-certificates action does not affect the container, so the
  container's trust store could otherwise still accept the compromised
  eMudhra/emSign intermediates.
@juenglin

juenglin commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 8ecac04

@juenglin
juenglin requested review from mdboom and rwgk November 5, 2026 16:22
@juenglin
juenglin enabled auto-merge (squash) November 5, 2026 17:37

@rwgk rwgk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had my agent look at both PRs together, asking it specifically to double-check the merge conflict resolutions. Here is what it found:

codex GPT-6.1-Sol ultra:

One confirmed omission in #3305; #3306’s conflict resolution looks correct with high confidence.

  1. [P2] #3305 needs PyYAML in its CI-tool test environment. ci-nightly.yml:45–54 creates an isolated environment and installs only pytest, but the newly backported driver-hash test imports yaml. I reproduced collection failing with ModuleNotFoundError. Add pyyaml==6.0.3, as main already does.

  2. #3305 also has a probable Windows coverage regression. coverage.yml:365–369 adopts standalone Python without the stack-reserve fix from #3342. Identical subprocess tests suffered stack overflows on main; the existing pytest worker-thread workaround doesn’t enlarge child-process stacks. This is strongly supported by the actual failure, but I haven’t reproduced it on this branch.

For #3306, comparing the original and rebased commits shows every hardening addition preserved, alongside the base branch’s Dependabot updates. I found no conflict-resolution defects.

Otherwise, the source-to-backport comparisons check out. Full CI passed at both reviewed heads (8ecac04 and b00b54e); local tooling tests passed 79 + 11. Regular PR CI doesn’t exercise the two jobs discussed above.

…ve fix

Finding 1 (ci-nightly.yml): The driver-hash test added by the eMudhra
backport imports `yaml`, which is not in stdlib. Add `pyyaml==6.0.3
pathspec==0.12.1` to the pip install in the CI-tools unit test step, matching
what `main` does. Without this, `ci-nightly` would fail at test collection
with ModuleNotFoundError.

Finding 2 (coverage.yml): Integrate NVIDIA#3342 ("Fix Windows coverage test stack
overflows") onto this branch. Add a `stack-reserve` input to
setup-hardened-python; both Windows coverage jobs (build wheel + test wheel)
request 8 MiB, letting the action patch the base interpreter with `pefile`
and verify the reserve at runtime. Switch the Windows pytest steps from
`run_pytest_with_stack.py` to direct `python -m pytest`, since the patched
interpreter now provides a larger stack on the main thread and in child
processes. The old MSVC `editbin` step is implicitly replaced by the action.

Identified in review by rwgk.
@juenglin

juenglin commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 455bb8a

@juenglin
juenglin requested a review from rwgk November 5, 2026 23:14

@rwgk rwgk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No new findings in 455bb8a. Both previous concerns are addressed correctly, with high confidence.

  • The corrected nightly dependencies pass all 79 tests in a fresh environment.
  • The Windows stack fix matches #3342 exactly and is enabled in both coverage jobs. Upstream Windows logs confirm the formerly failing subprocess tests pass with it.

@juenglin
juenglin merged commit fb8806e into NVIDIA:13.4.x Nov 5, 2026
219 of 222 checks passed
github-actions Bot pushed a commit that referenced this pull request Nov 6, 2026
Removed preview folders for the following PRs:
- PR #3305
- PR #3332
- PR #3343
- PR #3368
- PR #3380
- PR #3387
- PR #3388
- PR #3389
- PR #3391
- PR #3393
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI/CD CI/CD infrastructure cuda.core Everything related to the cuda.core module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants